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OBSERVATIONS WEIGHTED ACCORDING TO ORDER. 

By P. J. Daniell. 

1. Introduction. — When a series of measurements of some quantity are 
made, two particular quantities require to be calculated expressing re- 
spectively the norm and the deviation. For the norm the mean or the 
median is used while there are three measures of dispersion, the standard 
or root-mean-square deviation, the mean numerical deviation and the 
quartile deviation. The question is as to which of these are the more 
accurate under a general law. Moreover if we choose for our norm the 
mean or average it appears occasionally profitable to discard one or several 
extreme measures. Whether, or in what cases, this is legitimate is dis- 
cussed by Poincare* but no general conclusions are obtained. 

Besides such a discard-average we might invent others in which weights 
might be assigned to the measures according to their order. In fact the 
ordinary average or mean, the median, the discard-average, the numerical 
deviation (from the median, which makes it minimum), and the quartile 
deviation can all be regarded as calculated by a process in which the measures 
are multiplied by factors which are functions of order. It is the general 
purpose of this paper to obtain a formula for the mean square deviation 
of any such expression. This formula may then be used to measure the 
relative accuracies of all such expressions. 

Certain particular types are discussed and their accuracies calculated 
in percentages. 

Unfortunately the standard deviation is not of the same general type 
and therefore we add a note on its accuracy. The assumptions made are 
fairly general. On the one hand the number of observations, n, is supposed 
large and terms of order higher than 1/ra are discarded; on the other the 
probability law assumed is regular and indefinitely differentiable. In our 
applications to special types, however, we shall only consider cases n which 
the theoretical distribution is symmetrical, and this for logical reasons. 
It is useless to compare the relative merits of the various kinds of average, 
for example, the mean and the median, unless they all tend to coincide 
when n increases indefinitely. If there is a lack of symmetry both the mean 
and the median are necessary, or at least valuable, indications of the nature 
of the distribution. Indeed, in practise, their difference is sometimes 
regarded as a measure of lack of symmetry. 

* Poincare, "Calcul des Probability " (1912), p. 211. 

222 



Daniell: Observations Weighted According to Order. 223 

2. Mathematical Analysis. — Assume that n measurements t\, fe, • • • , t n 
are made and that their magnitudes are in the order of their suffixes, so that 

h Ss h, and so on. 

Multiply by the factors /i, ji, •••, f n , so that 

— n 

t = / , TrW. 
r=l 

We desire to find a formula for the mean square deviation of t when 
the measurements, tr, are subject to some law of probability p(f). 

If <p(ti, • • ',tn) is some function of the measures considered in their 
proper order, the average value of <p when t\, • • • , t n vary according to 
the law of probability will be denoted by Av (<p) to distinguish this from 
the weighted average, t, which we obtain for a particular fixed set of values 
t\> t%, • • • , t n . 

Allowing for the possible permutations of the suffixes, 



If 



P(tn)dt n I pitn-Jdtn-! • • • I p(ti)<p(t h '• • • , t n )dti. 

00 */— 00 *-' — 00 

Xp(t)dt = x, 
CO 

let t = t(x) ; then x varies from to 1, and 

Av (<p) = nl I dxn I dxn-i • • • I \ff(xi, • ••, x n )dxi, (1) 

Jo Jo Jo 

where 

\p(x U •••yXn) = <p[t(x{), ••-, t{x n )~]. 
We shall make frequent use of the formula 



I dx p I dx p -i • • • I f{x)dx = —. I f(x)(a — x) p dx. 
Jo Jo Jo P' Jo 



(2) 



This formula can readily be verified by differentiating with respect to a. 
A particular case is that in which f(x) = 1, 

£dx p £dx^ • • • £dx = ^jj(a- x)>1x = (^T)I aP+1 - (2a) 
• Substituting from (2a) in (1) 

Av (1) = nl I dx n I dxn-i ■ ■ ■ I dxi = nl • — -. • 1" = 1. 

Jo Jo Jo ^' 
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This confirms the coefficient nl in the formula (1). 

< dx n I dx n -i • • • I t(xr)dxi 

Jo «/0 

= n! I <fo» I dx^i • • • I t(x r )-, — L -Tr-.dx r [by 2aQ 
Jo Jo Jo v J- ) • ~ 

nl r 1 

= (n-rJKr-Dl J. * ( * )(1 ~ *>~*^ (3) 

When r, n are large the integrand will have a steep maximum near x = r/n. 
Also 

(n-^Kj-iyJ^^ x > xax n +ln + 2 n+p ' 

Denote r/(n + 1) by x r and neglect terms of order higher than 1/n. 

J" + 1 1 r + 2 2 

_^ = a;r + - ( l_ a;r ) > __ =a;r + _ (1 _ a . r)> etc . 

(n - r) ! (r - 1) ! J ^ ~ *>* ** ^ 

= « r OV + - (1 — Xr) • • • X r + ~ (1 — X r ) 



Xr "t" c\ $>r \*> $r 



•)• 



(n— r)l (r 



— jjj j (x- x r ) p (l - x) n - r x r -Hx 
= Sg!(p-g)! ( ~ 1)9a!? 



+ ,l?K^'- 1 ' > " }) l" g " 1) ^-^ 

Of these two sums the former is unless p = and the latter is unless 
p=2. 



nl 



—l)\j (* ~ ^ P(1 ~~ x) n - r x r -Hx = (p # 0, 2) 



(n — r)l (r 

= 1 (p = 0) 



= - av(l - av) (p = 2). 



[The reader is reminded that these equations are satisfied only as far 
as terms of order 1/n. J 
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Expand t(x) by a Taylor development near x = x r , 

t(x) = t(Xr) +(X- Xr)t'(Xr) + ~ ~^ t" (x r ) + • ■ •. 



Substitute into (3) and use the formula just obtained, then 

Av( 
By the same reasoning, 



Av (U) = t(x r ) + 2n X r (l - X r )t"(Xr). (4) 



Av (U 2 ) = f{x T ) + ~x r (l - x r )[2t(x r )t" (x r ) + 2{f (av)} 2 ] 

= [AV &)] 2 + I X r (l - Xr)[t'(Xr)J. 

IV 



(5) 



We next require to calculate Av (t r t a ) and must agree on order. Sup- 
pose s > r, then 

Av (trta) = n\ I dx„ I dXn-i • • • I t(x r )t(x s )dXi 
«/0 t/o t/0 

= (5-r-l)!(r-l)!j ^"J„ &n_1 

I *(*)&: I t(y)(x - yY^-y^dy (6) 

t/0 



(n -*)!(*- f- 1)1 (r- 1)! 

X (1 - *)"-*(a;)da:. (x - y)- r -y-H(y)dy. 

t/0 t/0 

In this double integral the integrand has a steep maximum near 

x = s\n, y = r/n. 

n' r 1 r* 

(n _, )I( .- f :. i)i (f -i)i j[ ^(i-«)-&J o vix-yy-'-y-vy 

_ r r + 1 r + g — 1 g+ g 3 + g + V — 1 

~ » + 1 ra+2* n+ q ' n+ q+ 1 n, + <7 + P 

Denote r/(re + 1) by x r , s/(n + 1) by x a and expand as far as 1/n, 
n~+2 * ^ + n ^ ~ ^' et °'' 
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n\ 



(n — s) ! (s — r — 1) ! (r 



x p (l - x) n ~*dx I y"(x - y)'- r - 1 y r - l dy 

- ** + qJ ^T ^ ~ x ^ + V{2q \l~ l) **** - *«> 

n 
Using a method similar to that given above, 

n! 



(n -s)l (s- r- 1)1 (r- 1)1 

X f (x- x s ) p (l - x) n ~°dx f (y- x r ) g (x - y)-^Y~ 1 dy = 0, 

Jo Jo 

except for p = 0, q = 0; p = 0, q = 2; p = 2; g = 0; p = 1, q = 1. 
In formula (6) expand 

<(«) = *(«,) + (sb - %)f (as.) H , %) = t(x r ) + (y- av)f(av) H , 

then by similar reasoning as before 

Av (UQ, (S> r) = t(Xr)t(x s ) + ^rO- ~ X r )t" (x r )t(x s ) 

+ £- X.0- - X s )t(x r )t"(x 3 ) + I 3V(1 - X s )t'(Xr)t'(x s ) (7) 

= Av (tr) " A-V (Q +-Xr(l — X B )t'(x r )t'(x s ). 
TV 

Now 

n 

t = 2^Jrh 
r=l 

Av(*) = E/,Av&), 



n n 



Av (?) = E/* Av (f r ) + 2 E E //. Av («.) 



» n 



[By 5, 7]. 



= E/?[AV &)] 2 + 2 E E /r/r Av (U) • Av (4) 



r — 1 IV 



n 7i 



+ 2E E/r/r ^- Xs) t'(Xr)i'(x.) 



Daniell: Observations Weighted According to Order. 227 

Let S 2 = n X % mean square deviation of (t) = n[~Av (f) — Av 2 (<)]. 
Then let 

f(Xr) = nf r , f(Xr)Ax = f(x r ) • - = /„ 

and replace the double sum by an integral. 

S 2 = 2 f /(x)(l - *)*'(*)& rf(y)yt'(y)dy. 

Jo Jo 

<p(t) = ffixitm, 

Je 

f <p(t)p(t)dt = 0. (8) 

J— oo 

<K*) = *(*)], V(x) = f(x)t'(x). 
S 2 = 2 f (l - *#'(*)& fynyWy. 

Jo Jo 

Consider the function 

F(a, 6) = 2 f (6 - 3)^(30(2* f (y - aW(y)dy, 

Ja Ja 

= 2 f y{x)dxC(y-aW(y)dy, 

dadb = ~ 2 jf *'(*)& jr^M = - W) - ^(«)] 2 - 

Integrating again and since dFjdb = 0, F = when a = 6, 
W f*r,,iA i/n-sj dF(0,b) 



Let 

where c is so chosen that 

Let 



dF 
db 

d 2 F 



db 



= [ ZW) ~ mJdy, ^M = jf [>(&) - mjdy, 

F(0,b)* f dx fdyOKx) - Hy)J, 
Jo Jo 



'0 «/o 

S 2 = F(fi, 1) 



= f dx fdyfyix) - t(y)J 

Jo Jo 

X+oo nt 

p(t)dt I p(u)du[<p(f) - <p(u)J. 
go J— oo 

Interchange the order of integration and also the symbols t, u. 

S 2 = f " p(t)dt C V {u)du\jp{u) - <p(t)J. 
J-x Jt 



iu 

12 
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Combining both forms, 

82 = 2 L n f_ a tv® ~ <p(u)Jp{t)p(u)dtdi 

/»+» r- s,+ca -i 2 

= J_ M ^®P® dt ~ [J_ <P(tMt)dt\ 
But by (8) the last term is 0; then 

S 2 = f + " <P 2 (t)p(t)dt. (9) 

This is the formula we set out to obtain. 

3. Norm and Deviation. — For the norm or average t = ^fA, with the 

r 
n 

condition 2/r = 1. 

r=l 

Expressing this by the approximate integral and then integrating by 
parts, 

f(t)p(t)dt = 1, 



f 

r*[.-p'(tn<p(t)dt = i. 

t/ — 00 



(10) 



The mean is obtained by equal weighting, f(f) = 1, <p(t) = t — t , where 
to is the theoretical average. Then 

S 2 = f " (t - t ) 2 p(t)dt = (T 2 . 

Then the mean square deviation of the mean of n measurements is 

S 2 /n = a 2 In. 

This particular result is well-known but it confirms our formula (9). 

If several groups of measures are to be combined, the average from 
each group should be multiplied by a factor inversely as the square of the 
deviation in that group. If then we agree to take the accuracy of the 
mean as a standard, equal to 1 or 100 per cent., the accuracy of any norm 
will be measured by the ratio <r 2 /S 2 . 

Definition. — The accuracy of a norm is defined to be (t 2 /S 2 , where a 2 
is the theoretical square deviation. 

In the case of the measure of deviation condition (10) no longer applies 
but we must suppose the weights / r chosen so that the average value of 
the deviation has a fixed value, D. 

Av(7) = £/ r Av&) = Z). 



j t (-tp)<p(t)dt [By (4)] 

+ 00 /»+» 
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Expressing in integral form, and integrating by parts, 

*J — 00 

-X 

= r L-t P '(t)-]<p(t)dt- f a P (tMt)dt. 

«7— co *J — CO 

Then, by (8), 

/ "[-i?'«)>P=D. (11) 

•y— oo 

For the measure of deviation condition (11) takes the place of (10). 

Again if we double the value of D, by doubling f r , we shall multiply 
S 2 by 4. A true measure of accuracy will be some multiple of D 2 /8 2 , and 
for reasons which appear later we make the 

Definition. — The accuracy of a measure of deviation is defined to be 
D 2 /(2S 2 ), where D is the theoretical average deviation. 

Standard Deviation. — The standard deviation may be defined as D where 

i » 1 / » \2 

D 2 = -ze-- 2 (T l tr) . 

n~i n 2 \7^ x r ) 

It is difficult if not impossible to obtain a formula, in the general case, for 
the average value of D; nevertheless, if the number n is large, the pro- 
portional error in D will be small of order 1/VrL We have the right, 
therefore, to assume that the proportional error in D will be one half that 
inZ> 2 . Then if 

D' = D 2 , 8' = n X mean square deviation of D', 

D 2 2D' 2 



2S 2 S' 2 
Choose the origin for t so that 



Let 



/•+00 

Av 00 = I tp(t)dt = 0. 

*J— co 

f fp(t)dt = a 2 , f " 'l*p(i)dt = q 4 



Then 

t r p(tr)dt r J •J p(ti) • • • p(t n )dti ••• dt n 

= J tp(t)dt = 0. 

Av (trQ = Av (W?) = 0, Av (t 2 ) = a 2 , 
Av (tt) = g 4 , Av {tr 2 t 2 ) = a 2 -a 2 = a\ 
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But 

1 " 1 / » \2 

The only terms in Z>' and D' 2 which yield integrals different from will be 
of the types f„ t\t\, t\. 

1 1 n — 1 

Av (5') = ~ M -^ (no*) = —^ o\ 

a /t^/2n 1 f a , ^C 71- 1) .1 2 r , , n n(n — 1) ,"1 
Av(D' 2 )=- 2 [^+2^-^^J-- 3 [^+2^ T -^^J 

j. 1 r 4 1 p »(»-d 4 i 



S' 2 



= n[Av CD' 2 ) - {Av (Z)')} 2 ] = ( —- 1 )V 



4 (n-l)(n-3) 4 



o". 



Omitting terms in 1/n, 1/n 2 , 

, 2 = 1 W ^^ = | 2 £^. (12) 

This formula gives the value of n times the mean square deviation of the 
standard deviation D. 

When the theoretical distribution is normal or Gaussian, 

„4 o^ C2 £> 2 3-l # 2 

By the definition, the accuracy will be 

D 2 

oo£ = 1 = 100 per cent. 

This explains the factor 2 which is introduced to make the accuracy of the 
standard deviation 1 when the law is normal. 

It is an interesting fact that the formula (12) proved for the standard 
deviation is the same as the corresponding value given by (9), when, instead 
of the mean-root-square deviation, we multiply every measurement by the 
theoretical value of t corresponding to its order in the series. For then 

f(t) = Xt, (p{t) = \\f + constant. 

This constant is chosen so that Av (<p) = 0, or 

<p(t) = i\(f - CT 2 ). 
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Using the condition which led up to (11), 

X<r 2 = D. 
From (9) 

& = * £' (f - o*)*p®di 

x %4 o4 . 4, fl y-* 4 

4. Jfefosi Accurate Weighting. — For the norm it is required to make jS 2 
given by formula (9) a minimum under condition (10). Then 

-«>-4-f§]. /.«>-4[-^§]- 

where from (10) 

1 r +co y 2 
Xi ./-« \v 

Accuracy 

2 /•+» /2 

For the deviation, S 2 given by (9) is minimum under condition (11). 

**(*) = x 2 (- £- 1), Mt) = x,^(- t^y 

where from (11) 

D r +m v' 2 X 2 

f- = fP-dt-1, S 2 = X 2 D = P 2 ^. 

X 2 J-00 P # 

Accuracy 

A =fH[£>7 <fi - 1 ]- 

For the normal law, 

p' t 11 Z> 



p a 2 ' Xi <r 2 ' X 



= 2. 



2 



/i(0 = i, Mt) = W, ^1 = i» ^ = 1. 

Thus for the normal law the most accurate norm is the equal-weighted 
average and the most accurate deviation that obtained by multiplying 
each measure by the algebraic theoretical deviation corresponding to its 
order. As we pointed out before the accuracy of the latter is the same 
as that of the standard deviation itself. 
For the symmetric Pearson law, 

_p[__ 2nt 
p ~ a 2 ±t 2 ' 

Mt) - 2\m (a2 ± f)2 , Mt) ~ (a 2 ± f) 2 • 
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If the distribution is supernormal, that is if the number of extreme cases 
is more than normal the weights for the norm and the weights -J- t for the 
deviation should diminish outwards and for the norm should even become 
negative for large values of t. 

On the other hand, if the distribution is subnormal the weights should 
increase and become infinite at the boundaries t = ± a. In these cases 
the weighting to be applied is much too complicated to be of any practical 
value, aside from the impossibility of knowing beforehand the proper 
values of a and n. However the most important cases are supernormal 
rather than the reverse and instead of letting the weights diminish according 
to a complex law we may take equal weights, for the norm, up to a certain 
point and then discard all measures outside these limits. Such a norm 
we shall call a discard-average and in practise a certain outer fraction of 
the measures is discarded. For the deviation we may discard not the 
outer but the inner fraction. Our next paragraph deals with such special 
types. 

5. Special Types of Average. — Discard Average. Let k be the central 
fraction of the group retained and let h be the solution of 



Then 



2 p(t)dt = k. (13) 

Jo 



= 0, otherwise. 



1 
P 



t> h 



By formula (9) 

S 2 = \ 2 [ 2 £ fp(t)dt + t\2 jf " p(t)dQ. 

Let a denote the ratio 2 I fip(t)dt : kt\, 

Jo 

rh 

2 t 2 p(t)dt = akt\, (14) 

Jo 

« 2 = |d- a -«)*:• as) 
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In the case of the median k and h approach together, and 

In the following discussion for purposes of comparison we shall use the 
normal Gaussian law and the two extreme Pearson symmetric forms, 

2 
supernormal, p(t) = — (1 -f- tf)~ 2 , (a) 

normal, p(t) = — p- e~ t2 , (b) 

3 
subnormal, p(t) = | (1 - ?), (f^l). (c) 

The accuracy of the median will be, in percentages, 

(a) 162, (b) 63.7, (c) 45. 

For the quartile-discard average, k = 1/2 and the fraction a lakes the 
values, 

(a) .306, (b) .314, (c) .323. 

Formula (15) becomes 

S 2 = 2tf(l + a). 

The accuracy will be, in percentages, 

(a) 200,* (b) 83.7, (c) 63.* 

In a supernormal Pearson distribution, 

p(t) = c(l + f)~\ 

For the quartile-discard average, when n is large, 

1.195 r .851 .700] 1.195 

* 2n |_ n « 2 J 2n- 1.70* 

Hence the accuracy of the quartile-discard average will be 

a „„„2n- 1.70 „„„ , 109 

A = 83J 2»-3.00 per CenL = 83J + 2^"3 per Cent ' 

The formula can hardly be used with accuracy when n is as small as 2, 
but even then it would give the value 

A~= 193. 



' These values are only rough approximations. 



234 Dantell: Observations Weighted According to Order. 

The quartile-discard average will be more accurate than the ordinary mean if 

2n < 9.3. 

If <r 2 is average ft and q 4 average ft then the above condition may be trans- 
lated into 

<f > 4-4<r 4 , 

instead of 3<r 4 for a normal law. 

Median-Quartile Average. — This average is the mean of the median 
and the two quartiles, 

«-|(«, + m + «,), g >_4(I, + JL + !), 

where p — p(to), Pi = p(f\) and h is the theoretical quartile deviation. 
For a normal law the accuracy is 

A = 86.0 per cent. 

It appears to be a little more accurate than the quartile-discard average, 
but we have assumed that the number of observations is large. When the 
number is small it will be difficult to determine the quartiles exactly, so 
that, taking everything into consideration, we may say the median-quartile 
average and the quartile-discard average are about equally accurate. 

The most serious objection to the use of any special type of average is 
that discontinuity is introduced; that is, if the measures are considered 
as sufficiently normal, none will be discarded; if not, some may be discarded 
and there will be a finite change in the average. To obviate this difficulty 
we might use a combination of quartile-discard and ordinary average. 

Let p be the weight assigned to the ordinary average. 

q = 1 — p = weight assigned to the quartile-discard. 
c 2 = mean square deviation. 
N = mean numerical deviation. 



P = quartile deviation, or probable error. 



Then 



52 = pV + 2pqP(2N - \D) + f | P 2 , (16) 

approximately, assuming average ft from to P is \P 2 , average t from 
to P is \ P. Then we may choose p, q so as to make S 2 minimum. 

6. Special Types of Dispersion Measure. Numerical Deviation. — In this 
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case 

f(t) = + 1, t> 0, 
= - 1, t < 0, 

<p(t) = \t\ - N, 

where N is the theoretical mean numerical deviation. Then, by (9), 

£2 = „* - N 2 . 

By the definition, succeeding formula 11, the accuracy will be 

A 2 - N2 

2S 2 ~ 2((7 2 - N 2 ) • 

In the case of our three Pearson types (a) supernormal, (6) normal, (c) sub- 
normal, the accuracy, in percentages, will be 

(a) 34, (b) 87.6, (c) 118. 

Discard Deviation. — In this case we discard the inner portion and then 
use the mean numerical deviation of the remainder. Under a normal law, 
if the portion between the quartiles, that is the central half, is discarded 
the accuracy is 96.3 per cent. Hence this is practically as accurate as the 
standard deviation and may, in some cases, be more rapidly found as it 
is a numerical mean and the calculations are made for half the measures only. 

Quartile Deviation, etc. — If t is the theoretical deviation the accuracy 
will be 

8[tp(t)J. 

For the quartile deviation this is 36.7 per cent. 

It will be a maximum when tp(t) is maximum and for a normal law this 
is given by t = a. The values t = ± <r practically divide the whole range 
of measures into two thirds within and one third without. 

We may call this the sextile deviation, remembering that it is the 
outermost sextiles only which are used. Then the accuracy becomes 
46.8 per cent. Furthermore it is much easier to find the corresponding 
standard deviation a in this case, for theoretically with a normal law 



-0 + s) 



outer-sextile deviation. 



We might also call this the semi-probable error, that is, such that the 
chances of exceeding it are just one half the chances of not exceeding. 

A table is added of the accuracies of the various types when the law 
is assumed to be normal. 
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Norms : 

Median 63.7 per cent. 

Quartile average (Qi + M + Q,)/, 86.0 " " 

Quartile discard average 83.7 " " 

(outer quartiles discarded) 
Mean 100 " " 

Dispersion: 

Quartile deviation 36.7 " " 

Outer sextile deviation 46.8 " " 

Numerical deviation 87.6 " " 

Discard deviation 96.3 " " 

(inner quartiles discarded) 

Standard deviation 100 " " 

Rice Institute, 

Houston, Texas. 



